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INTRODUCTION 



1. Purpose Of the Manual 

One of the major ooais of college education is teaching the content of a 
discipline in such a way mat students can see relationships among concepts and 
facts. You. as an instructor, hope that students will leave your courses with more 
than mere facts. You hope that they will develop a mental frameworK or 
conceptual structure that will enable them to remember and apply the facts and 
theories they have learned. And you hope this framework will establish a solid 
basis for continued learning. To help students develop such a conceptual 
structure you need to know what they already know and what they are gaining 
from your course, and what they take away at the end of ihe course. 

This manual describes two methods of assessing conceptual structures. 
By "conceptual structure" or "knowledge structure" we mean a set of concepts 
that are or may be related to one another. Conceptual structures are the 
frameworks into which new phenomena and facts may be fitted. 

Teachers have long recognized the importance of teaching students to 
look for relationships, out we seldom have had good ways of assessing the 
degree to which students learn appropriate conceptual relationships. The 
methods that follow are tools teachers and researchers can use to assess student 
learning of conceptual relationships. 

While cognitive psychologists have developed other methods to assess 
concept development in laboratory experiments, few are adaptable to classroom 
situations (McKeachie. Pintrich, Un, Smith & Sharma, 1990). The "ordered-tree" 
and "fill-in-the-structure" methods we describe in this manual have been 
successfully used in college ana university classes. 



2. Uses of the Techniques 

When students enroll in a course, we usually assume that they come with 
some prior knowledge. This level of knowledge at the beginning of a course, and 
especially the organization of that knowledge, is in many cases only assumed by 
the instructor. The students' initial knowledge can be accurate ana relevant, 
reflecting what has been learned previously. More often it is inaccurate, reflecting, 
among other things, stereotypes prevailing in our society [see Box A]. Having 
information about students' initial knowledge should help you develop materials 
for the course, improve organization of the different topics, and stress the issues 
about which misconceptions or stereotypes exist. The techniques described in 
this manual are designed to help you assess students' knowledge and 
understanding. They can also help student diagnose the difficulties they may 
have in learning. Simply listening to a lecture or reading a textbook will not be 
sufficient for understanding. To become aware of the connections between 
concepts and facts students need to develop the same cognitive structure as that 
of the teacher or textbook author, presumably a structure representing the 
discipline. 



Box A 

We have used the techniques described here to assess students' initial 
knowledge at the beginning of a Psychology of Aging course. We asked the 
instructor to select 10 to 20 concepts that were important for students to 
understand in the course. This instructor chose concepts Including 
"attachment," "intimacy," "encoding," "retrieval," and "mid-life crisis,^ among 
others. Students were then given this list of concepts and v^re asked to list 
them sc that those most alike appeared together, the results of these student 
trials were then tabulated to provide a picture of the students' cognitive 
structures, the frameworks students use to understand and relate concepts. 

When kx>king at students' organization at the beginning of the course, 
across all students in the class, we were able to identify organization based on 
three separate sources. First, these structures reflected some prior, general, 
common-language understanding of the relationships among the concepts. 
For example, "attachmenr and "intimacy" tended to appear together, reflecting 
the fact that these two concepts are associated in everyday life. 

Second, the initial structure also represented knowledge gained in 
previous psychology courses. This was illustrated by the frequency of the 
cluster containing "encoding" and "retrieval," two concepts that appear 
together in current psychology. 

The third type of knowledge represented by the initial structure was 
misconceptions students have about the subject matter. Their belief in steep 
decline of abilities during middle age (a decline not supported by evidence) 
was indicated by the cluster containing the concepts 'mid-life cnsis" and "wear 
and tear." 



A second type of information needed to improve teaching and evaluation is 
how students' knowledge structures develop throughout a course. Pre- and 
post-tests given in courses typically show that students learn knowledge and 
concepts. But in many cases they don't tell us about the way the organization of 
facts and concepts is developed. Measures revealing the development of student 
conceptual structures can help you adjust your teaching to students' current 
conceptual stnjctures (e.g., by reviewing some of the materials and stressing 
some parts that seem poorly grasped) and develop explanations that bridge the 
gap between students current understanding and your goal. 



gpxg 

In the Psychology of Aging course, students' structures in 
the middle of the course reflected both the relationships among 
concepts learned in the first half of the course and also some 
breakdown of misconceptions they had at the beginning of the 
course. For example, by the middle of the semester, fewer 
students thought that "mid-life crisis" was closely related to "wear 
and tear." 



Finally, assessing students' knowledge at the end of the course could help 
you evaluate the course's outcomes In relation to your original objectives [see 
Box C]. 



In the Psychology of Aging course, mentioned above, the 
prior goal of the instructors was to stress both topical (biological, 
cognitive), and developmental (young, middle, and late 
adulthood) dimensions, yet students'^structures at the end of the 
course revealed much more awareness of the topical than of the 
developmental dimension. In retrospect, although we intended to 
combine these two dimensions, in practice it proved difficult. For 
example, it was hard to discuss cognition in young adulthood 
without comparing young adulthood cognition to that of middle 
and old age. As a result, the course actually presented each 
topic independently of a given stage of development. Information 
about students' knowledge structures was useful to us as we 
tried, in the following semester, to better integrate the two 
perspectives. 



Typically, we assess course outcomes by achievement tests, which 
measure knowledge of facts and concepts, but most tests provide little 
information about the students' organization of concepts and facts. The 
techniques described here will help you determine how those concepts are 
related to each other. 

To summarize, the techniques suggested here can help you assess your 
students' knowledge organization at the beginning of a course (or a unit within the 
course), during the course, and at the end of the course. In addition, your 
students can analyze their own structures. 



3. Modes of Assessment 

The techniques described here, the "ordered-tree" and the "fill-in-the- 
structure," have been used in science, social science, humanities, and other 
courses by instructors and students in liberal art colleges, community colleges, 
comprehensive universities, and major research universities. 

Both the "ordered tree" and the "fill-in-the-structure" methods can be 
administered to classes or to individuals for diagnosis of prior knowledge or 
evaluating achievement for a class or for an individual student. In-class 
administration should not take more than one class period for either of the 
techniques. 
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4. .Mfiasucfis 

Both the 'ordered tree' and the "filUin-the-structure" techniques prov'de 
several measures of student organization of the material. For example, the 
ordered tree technique enables you to characterize a student's organization of 
the material in four ways: 

a. The anr junt of organization the student has (reflected by the number of 

concepts that are grouped together consistentiy by the student). 

b. The number of different hierarchical levels along which a student 

organizes the concepts. 

c. The similarity of a student's organization to that of the instructor. 

d. The direction of relationships between concepts as perceived by the 

student. 

The fill-in-the structure technique enables you to characterize a student's 
organization of the materials in two ways: 

a. The degree to which a student has an organization of general and 

specific concepts. 

b. The departures of a student's organization from that of the ins: uctor. 

Here, either global or specific departures can be assessed. 
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I THE ORDERED-TREE TECHNIQUE 



A. Preparation of the Stimuii 

The ordered-tree technique is named for the tree-like form the analysis of 
results takes: basic concepts form the Injnk" and "tsranches' and detailed 
concepts make up the "leaves " To use this technique, follow these steps: 

1 . Rrst, choose a set of concepts representing the subject-matter to be 

assessed. One way to develop your structure of the materials is by 
listing as many concepts as you can think of in the course (e.g., all 
animals names). 

2. Then group the concepts into several categories (e.g. birds, fish, 

mammals). 

3. Next, arrange the concepts in each category to reflect their 

relationships. For example, if one concept is an example of another 
one, they should be connected hierarchically where the first concept 
is below the second (e.g., a lion is an instance of mammals; a catfish 
is an instance of fish). 

4. After you have organized each category, put all the groups together 

under the top of the hierarchy, which includes the course title. You 
can make any changes now, until the structure is compatible with 
your view of teaching the course. This is the instructor's structure 
(see Figure 1 for an example for a Psychology of Aging course). 

5. At the end of this process you should have a stnjcture with 16 to 20 

concepts at the lowest level . These will be the concepts given to the 
students as part of their task, as described below. 

These concepts should represent major content areas in the course (or 
unit of instruction), with a few concepts from each area. The concepts should be 
from different levels of generality. These concepts will probably reflect materials 
from lectures, the textbook, or both. Your final structure should have at least two 
related concepts In each group or "chunk" and the various branches of the 
structure should not be connected at the lowest levels. 

Note that the final list given to students does np{ include major abstract 
concepts. In the example from the Psychology of Aging course, "stages of the life 
cycle," a major focus of the course, is in the middle of the structure with the terms 
"young adulthood," "middle adulthood," etc. 
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Figure 1 instructor's Knowledge stnjcture: Psychology of Aging course 




The final list given to students includes issues/tasks associated with each stage. 

Another way to create the structure is by using an empirical approach. 
After choosing the concepts, you can actually perform the ordered-tree task 
described below. Analyzing your results will provide you with your structure of the 
materials. 



B. Materials for the Task 

Prepare a booklet for each student. This booklet should include four or 
more pages. Each page will have the same set of concepts (an even number 
between 14 and 24, as required by the computer program that analyzes the 
results) that are at the lowest levels in the instructor's structure. The concepts 
should appear in rows and columns in no specific order. For example, for 16 
concepts you might arrange them in 4 rows and 4 columns. The order of the 
concepts in each page should be different to prevent students from simply 
repeating the same order on each trial. 

On the right side of each page should be vertical blanks corresponding to 
the number of concepts that appear in the rows and columns. In Figure 2 you 
can see an example of the four pages taken from a Psychology of Aging course. 

The first and fourth pages should instruct the students to begin with any 
word they may choose (uncued trials). On the second page, choose any word 
you wish (perhaps randomly) and ask the student to begin with that word. For 
the third page, choose another word as the beginning word. Thus, the second 
and third trials are "cued" trials. 



0. Administration 

1 . Ask students to arrange the concepts in a vertical order so that 

concepts closely related in terms of their meaning in the course 
appear close to each other. In the first trial students start with any 
one of the concepts (an uncued trial). Our experience shows that 
students usually need between 5 and 8 minutes to perform the task 
It is important to stress that they use every concept, and each only 
once. 

2. When the students have completed the first page, lecture, or assign 

some other activity, for a few minutes (4-6), then ask them to fill out 
the second page of the booklet under the same instructions. 



Figure 2 Examples of cued and uncued trials 
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this time starting with a specified concept (cued trial). They should be 
given about the same time to complete the task as they were given 
for the first page. 

3. After a few more minutes of activity unrelated to the ordered-tree, have 

the students turn to the third page and ask them to perform the same 
task as described above, this time starting with a different concept. 

4. After another interval of unrelated activity, students should turn to the 

fourth page and be asked to relate the concepts, this time starting 
with any concept they wish (uncued trial). 

It should be noted that repeating the procedure four times is reasonable 
when using this number of concepts. In some of our studies we created 
structures based on three or five trials. Three trials showed much less structure in 
comparison to the four-trials case, whereas five trials added little. As a result we 
recommend using at least four trials. If there is enough time, more trials can be 
used (while alternating between cued and uncued ones). Figure 2 presents an 
example of cued and uncued trials. 



D. Scoring 

The four trials (or more, depending on how many you actually use) for 
each student are coded using an IBM-PC computer program that will summarize 
a student's responses in both a formal expression representing a tree structure 
for that student, and a graphic representation of this student's linowledge 
structure. As an example, Rgure 3 shows the four strings produced by a student 
in a Psychology of Aging course, with the resulting computer output. Information 
in parentheses indicates chunk, concepts that consistently appear together in the 
student s strings. (For information about how i: ;e program works and the 
assumptions underiying it, see Box D and the article by Reitman & Rueter 1980) 
The instructions about using this program are in the file READ ME.TEXT on the 
disk. 
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Figure 3. An example of a student's trials and resulting 
computer expression and structure. 

Student's Trials 

1. sen wea aut pre rea enc ret sel Ing int att soc lif ber mid ere 

2. ere ing sel lif enc ret sen rea wea aut pre soc att int ber mid 

3. sen enc ret rea wea aut pre sel ing ere I1f att int soc ber mid 

4. int att soc ber lif sel ere ing mid rea wea aut pre sen enc ret 



Computer Expression 
(<<sen<[wea aut pre] rea>>[enc retj> sel ing<<int att> soc> lif ber mid ere) 



Computer Structure 



Aging 
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The program is based on a theory of mental organization that assumes 
that single concepts or sets of concepts are mentally organized into a 
hierarchy whose lowest level terminal nodes represent single concepts and 
higher-level nodes represent more abstract categories or concepts. The 
technique capitalizes on people's tendency to list all items in one chunk of 
information before moving on to the next chunk. From a set of cued and 
uncued trials, the algorithm efficiently finds the set of all chunks for each 
subject and represents this set as an ordered-tree that represents a student's 
knowledge structure from a course. 



For each student, four measures are provided by the computer program. 

1 . Amount of organization : This measure, Possible Recall Orders {PRO)\ 
provides information about theMlfiUOJ of organization in a student's structure. If 
the order in which a student writes the concepts in each trial is completely 
different, it indicates a small amount of organization of the materials, since he or 
she relates different concepts to each other in each trial. However, if a student 
uses a consistent order of concepts across trials, it indicates a high degree of 
organization. 

The computer program accompanying this manual computes the amount 
of organization for each student's responses. Note that a large number of 
Possible Recall Orders (PRO) indicates little organization, and a small PRO 
indicates high organization, i.e. the students group concepts together 
consistently. The range of PRO scores varies according to the number of 
concepts used. For example, for 14 concepts the range is from 0 (very organized 
structure) to 36 (unorganized structure); for 16 concepts PRO scores range from 
0 to 44; for 18 concepts from 0 to 52 and for 20 concepts from 0 to 61 . 

2. IJgptti: Another measure shown by the "ordered tree" is the average 
hierarchical depth* (Average number of levels in each chunk). Larger average 
depths indicate more structure and differentiation among concepts in terms of 
their hierarchical position. A depth measure of 1 .0 indicates no depth at all. The 
chunks "Dogs-Wolves, Cats-Tigers, Rattlesnake-Cooperhead," would represent 
three levels (1-dogs, etc; 2-mannals; 3-animals), resulting in a depth measure of 
3.0. The chunk, "Cat-Catnip," has only one level resulting in a depth measure of 
1 .0. The higher the depth value, the more elaborate is the structure. Use the 
computer program to compute a depth measure for each subject. 

3. Similarity : This measure provides an index of similarity between a 
student's stmcture and that of the instructor (or between any two students' 
structures). Generally, the greater the number of chunks a student's structure 
shares with that of the instructor, the greater the index of similarity (see 
McKeithen, Reitman, Rueter, & Hirtle, 1981 for details). A high value on this 



1 PRO is the natural logarithm (base 2) of the number of different written orders 

that could contain its chunks. 

2 Depth is defined as "the average number of nodes between root and terminal 

items." 
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measure indicates similarity in a student's cx^nitive structure to the structure of 
the instructor, or the degree to which the student has learned what the teacher 
taught. This measure ranges from 0.0 (no similarity) to 1 .0 (structure identical to 
the instructor's). Again, to obtain this measure, use the computer program. 

4. Directionality : A cluster of concepts can be non-directional, 
bidirectional, or unidirectional. A non-directional cluster [indicated by 
parentheses ( )] in the stnjcture obtained by the computer, is one in which three 
or more concepts occur together consisteritly, but in no consistent order; a 
bidirectional duster (indicated by arrow brackets < > ) is one in which the 
concepts in the cluster are consistently ordered but the student may start from 
either end of the duster; a unidirectional duster (indicated by square brackets [ ] ) 
is one in which the duster is consistentiy written by the student in one order. 
Thus, the directionality measure seems to indicate the direction of relationships 
between concepts as perceived by the student. A concept that is first accessed 
in a cluster may be more salient or more accessible in memory than those coming 
later, or may precede the others chronologically, as in the chunk "Stone age - Iron 
age". 



E. Interpretation of the Results 

Students who have high scores on all the measures have appropriate 
conceptual structures. Those who have a large amount of organization and 
depth but a low similarity index, may organize the materials studied, but their 
organization does not fit the instructor's. These students are likely to misinterpret 
the instructor's explanations and to distort facts to make them fit their own 
structure. They wtii need opportunities to see how their misconceptions lead to 
errors and opportunities to develop more realistic structures. Students with both 
little organization and low similarity need to work on learning the materials in the 
course, on looking for relationships between concepts, and on developing a 
general tendency towards integrative, conceptual learning. 
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II. THE FILL-IN-THE-STRUCTURE TECHNIQUE 



In the "fill-in-the-structure" teuinique the Instructor's structure is presented 
with some concepts omitted. The student's task is to fill in the blanks. 

A. Preparation nfth^ft Mmftflfllff 

As a first step, the instaictor should come up with a hierarchical 
representation of the subject-matter domain to be taught to the students This 
means, as in the "ordered tree" technique, graphic representation of major 
concepts and the relationships among them. An example of such a 
representation from an ecology course appears in Rgure 4. The number of 
concepts to be used is limite i only by the time It will take the students to do the 
task. We have used structures with 30 to 55 concepts. 

The riext step involves taking out some of the concepts, leaving their 
spaces btenk These concepts are then placed at the bottom of the paae 
intermixed wrth concepts not used in the structure (distractors). We have used 
♦hl^lSSL ^ distractor concepts. Usually we recommend that no more 

than 50% of the concepts be taken out of the structure. In addition, you should 
Ijy h ^ ^^"^^ percentage of concepts from each level of the 



B. Administration 

1 . Students receive the hierarchical graphic representation of course 

materials in which some of the concepts are missing. (See Figure 5 
for an exarnple in the ecology course). These concepts appear at the 
bottom of the page intermixed with distractor concepts. 

2. The students' task is to fill in the concepts in the appropriate position in 

the structure while recording down thp nr^^f > which thay ^ .cinn 
the concepts. The students will usually nee^^^ 15 and 25 

minutes to complete this task, depending on the number of missing 
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Figure 4 



Instatctor's Knowledge Structure: Ecology Course 
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Figure 5 FilMn the-structure in Ecology course 
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Several measures can be obtained for each student. These measures 
include: 



1 . The total number (or percentage) of correct responses. This indicates 

the concepts that were correctly filled in, according to the instructor's 
structure. This measure provides an overall similarity index of a 
student's cognitive structure to that of the instructor. 

2. The number of concepts that were placed correctly at each level of the 

hierarchy. These measures provide information about the similarity 
between each level of a student's structure (from the upper, more 
general, to the lower, more specific) and that of the instructor. 

3. The order in which the concepts were filled in. This describes how 

students access the structure. For each level of the hierarchy add 
the number of the orders the students listed when filling in the blanks. 
Divide this sum by the number of blanks filled in. Smaller average 
orderings at lower levels of the hierarchy imply that students access 
their structure in a bottom-up fashion (from the specific to the more 
general concepts). In contrast, smaller average orderings at the 
upper levels of the hierarchy imply that students access their 
structure in a top-down fashion (from the general to the specific). 

4. A final group of measures can be obtained by looking at theiyo^f 

errors that students committed. For instance, two students could 
misplace the same number of concepts but at different places in the 
structure. 

a. One type of analysis of types of errors, looks at the degree of 

misplacement. For example, you can compute a measure of 
distance between the blank in which a concept was placed and 
the blank where it should have been placed, moving along the 
shortest path which connects these places, and then average 
these distances across all errors made by a student. A high 
value on this measure indicates significant deviations from the 
instructor's organization. See the Appendix for an example of 
the scoring method. 

b. Another type of analysis looks at whether misplacements occurred 

within the same chunk or between chunks. Total the errors 
within chunks and the errors between chunks. Errors within a 
chunk may indicate some similarity to the instructor's structure, 
while errors between chunks indicate some major structural 
differences. 

c. Similarly, errors can occur within the same hierarchical level or 

between different levels. Total the errors within levels and the 
errors between levels. Errors within the same level indicate 



some similarity to the instaictor's structure in terms of the 
hierarchical relationships, while errors between levels indicate 
hierarchical dissimilarity. 

d. Rnaily, analyses of erors can be done to assess relationships 
between concepts. For example, ail sets of "parent-child" 
relationships (e.g.. the related concepts "Memory" [higher-level] 
and "Retrieval" [lower-level] in Figure 1) in the instructor's 
structure could x>e scanned to reveal their placement in a 
student's structure. For example, a parent and a child that were 
both misplaced by a student but that kept their hierarchical 
relationships intact, indicate knowledge about these 
relationships. Alternatively, a parent and a child that were both 
misplaced by a student and placed as "siblings", may indicate 
that a student has information atx^ut their being related but does 
not know the type of relationships between them. Rnaily, a 
parent and a diild that were misplaced in different parts of the 
structure may indicate that the subject does not know that they 
are related. 

An example of how to calculate these measures of performance appears in 
the Appendix. 

The analysis can also be used on specific parts of the structure if a 
particular topic represented in the structure is of interest. As you can see, there 
are various options for scoring in tiiis task, and you may think of others that are of 
interest to you. Analyzing their own structures can be a useful learning 
experience for students. 



0. Interpretation of the Results 

A student with a small percentage of correct responses needs to work on 
gaining a better understanding of the concepts and their relationships. If more 
errors occur at the upper levefof the structure, the student needs to focus on 
overall organization. If, however, errors primarily occur at the lower levels, 
students need to focus on basic definitions and examples and the relationships 
between these concepts and more general ones. 

If many errors occur between hierarchical levels, you can help students by 
reviewing the hierarchical relationships between the concepts. Errors in specific 
parent-child or sibling relationships can be corrected by reviewing the correct 
relationships between the relevant pairs of concepts. 

Small group discussions in which students compare their structures can be 
useful in helping students to develop more accurate and complete structures. 



* Parent = higher level; Child = lower level 



III. CHOOSING THE RIGHT TECHNIQUE 



To help you choose which technique to use. we present some considerations of 
similar and different aspects of the ordered-tree and fill-in-the-structure 
techniques. 



A. Similarities: 

1 . Both techniques use the notion of a knowledge structure represented as 

a group or concepts interrelated in an hierarchical fashion. 

2. Both provide several manifestations and indicators of a student's 

structure. 



B. Differences: 

1 . The ordered-tree indirectly infers structure, because we infer the 

student's structure indirectly from his or her orderings of the concepts. 
The student might not necessarily be aware of the hierarchical nature 
of the relationships. The fill-in-the-structure. however, is a direct 
technique where the student is aware of the hierarchical representation 
of the materials. 

2. The ordered-tree technique is less flexible in that it results in strict 

hierarchical relationships. The fill-in-the-structure technique allows 
deviations from hierarchical relationships (e.g., two parent concepts 
can be both related to a mutual child). 

3. The ordered-tree takes longer to complete. 

4. In the ordered-trrie, the subject is less constrained by the type of 

relationship? he or she may show, while in the fill-in-the-structure, the 
subject's responses are constrained by the graphic representation, 
which includes some predetermined relations. 



C. Order of Administering the Two Techniques: 

Both tasks can be administered to the same class, although it is bet.ar to 
perform the ordered-tree first to minimize the practice effects from one task to the 
other. There is little information in the ordered-tree task to serve as cues for the 
fill-in-the-structure task. Usina the fill-in-the structure task first, however, could 
reveal to the students parts of the instructor's structure, hence affecting the order 
in which they list the concepts in the ordered tree task. 

We have established the reliability and validity for these measures in 
several research studies (Naveh-Benjamin, McKeachie, Lin, & Tucker, 1986; 
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Naveh-Benjamin, McKeachie. & Lin. 1987, 1989; Naveh-Benjamin, Un, & 
McKeachie, 1989). The various measures extracted from both techniques were 
shown to be related to course performance and to develop substantially 
throughout various courses. Although they have been used successfully in a 
variety of courses, the sort of hierarchical structure which these techniques 
measure is probably more cleaHy represented in science courses than in 
humanities courses (Donald, 1986). 
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APPENDIX 

An example of scoring thft Fili-in-the-stmcture task 



This example refers to the performance of a hypothetical student in a Learning to 
Learn course, where the student's responses are compared with those of the 
instructor. The structures of both appear on Rgure 6. 



a. Total numb er of correct responses (percentage correct)- 
7 of 16 = 43.75% 

b. Number of concepts plac ed correctly at each level of the structure : 



Level 1 


- 2of3 


= 66.7% 


Level 2 


- 2of2 


^ 100.0% 


Level 3 


- 1 of 3 


= 33.3% 


Level 4 


- 2 of 8 


= 25.0% 



It can be noted that this student had fairly accurate organization of higher 
levels of the structure (major concepts) but not as much organization of 
lower levels of the structure. 

c. Order : 

Average at Level 1 [1+2+3=6] 6 / 3 = 2.0 
Average at Level 2 [4 + 5=9] 9/2=4.5 
Average at Level 3 [6+7+8=21] 21/3=7.0 
Average at Level 4 [9+10+11+12+13+15+16+14 = 100] 
100/8 = 12.5 

Note that this student, in filling in the structure, completed it in a top-down 
direction. The average order is the lowest for the upper level of the structure 
and it gradually increases as we get to the lower levels. 



Figure 6 Th« instructoi's and Studenrs Structures: LBUrimQ to Learn course 
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d. Types of Error : 

(1) Degree of misplacement. 

For each error we compute the shortest distance between the 
expected placement (according to the instnjctor's structure) and the 
one obtained by the student. Then we average these distances 
across all the misplaced concepts. 

For example, "chunking" was misplaced by a distance of 6 (going 
from its observed placement through, "attention," "learning to learn," 
"memory systems," "working memory," and "limited capactty," to its 
correct placement). 

We can calculate the distance for all concepts misplaced in the came 
way. 

chunking = 6 
primary task - 6 
recall = 1 
episodic = 6 
scripts = 3 
incubation = 1 
planning = 6 
problem solving = 1 
retrieval = 1 

The average misplacement distance is 31 / 9 = 3.44 
(2) Misplacement within and between chunks. 

For convenience, let us consider as chunks the four major branches 
in the instructor's tree. 

# of within-chunk errors = 5 

# of between-chunk errors = 4 

In this case there were about same number of between-chunks and 
within-chunks misplacements. 
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(3) Misplacements within and between levels. 

# of within-level errors = 3 

# of between-level errors » 6 

This student shows twice as many between-level misplacements, 
possibly indicating confusion between general and specific concepts, 
especially in levels 3 and 4, where most en-or occurred. 

(4) Errors in relationships between concepts. 

Following is an example of an analysis of parent-child errors. 

There were six such relationships in the instructor's structure to fill 
in (e.g., learning strategies-metacognltion). 

* Of those, two were con-ect in the student structure. In other words, 

both parent and child were placed in their appropriate positions, 
(e.g. memory systems-working memory). 

* Of the four parent-child relationships that did not hold, two were 

reversed (the child became the parent and vice versa, e.g., recall- 
retrieval). 

One set was placed as siblings (semantic-scripts). 

One set drifted apart altogether (metacognition-planning). 

It may be noted that in three of the four errors of this type, the 
association between the pair members was kept (they still were put 
close to each other). However, what has been missed is their 
hierarchical relationships. This means that this student could not 
identify the correct relationships between the concepts, implying 
some loss of their hierarchical organization. 
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Attachment 



IBM PC disk for analyzing the ordered-tree task. 



If you have any questions about the use of these techniques in your own classes, 
please write to: NCRIPTAL, 2400 SEB, University of Michigan, Ann Arbor, 
Michigan 48109. 
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